Faces in Event Streams (FES): An Annotated Face Dataset for Event Cameras.

The use of event-based cameras in computer vision is a growing research direction. However, despite the existing research on face detection using the event camera, a substantial gap persists in the availability of a large dataset featuring annotations for faces and facial landmarks on event streams, thus hampering the development of applications in this direction. In this work, we address this issue by publishing the first large and varied dataset (Faces in Event Streams) with a duration of 689 min for face and facial landmark detection in direct event-based camera outputs. In addition, this article presents 12 models trained on our dataset to predict bounding box and facial landmark coordinates with an mAP50 score of more than 90%. We also performed a demonstration of real-time detection with an event-based camera using our models.

Angular Photogrammetric Analysis of Facial Soft Tissue by Image Processing Algorithms.

BACKGROUND: The main aim of this study was to present an automatic method based on image processing algorithms for facial anatomical landmark localization and angular photogrammetric analysis applicable for rhinoplasty surgery. We studied and measured color profile photographs of 100 patients before and after rhinoplasty surgery. METHODS: In facial anthropometry analysis, anatomical landmarks are often defined by specialists, manually. This process is time-consuming and requires training and skill. The Cascade Regression Method (CRM) was utilized for facial landmark detection to overcome the mentioned problem. In this study, 11 anatomical landmarks were used to measure 9 facial angular metrics. Finally, a t-test (with the significance level set at a p-value of 0.05) was applied to analyze before surgery versus after surgery comparisons. RESULTS: Experimental results dedicated that there is a significance difference (p < 0.001) in nasofrontal, nasolabial, mentolabial, nasomental, facial convexity including nose, facial convexity excluding nose, projection of the upper lip to chin, and H angles before and after surgery. Also, results showed that there is not a significance difference in nose tip angle. CONCLUSION: We believe that the presented system can aim to reduce the personal errors made by manual measurement and to facilitate facial anthropometry analysis before and after surgery with high accuracy. Also, the normative data for Iranian women can be used as a guide for the diagnosis and planning of oral and maxillofacial, ENT, and plastic surgeries. LEVEL OF EVIDENCE II: This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

A store-and-forward cloud-based telemonitoring system for automatic assessing dysarthria evolution in neurological diseases from video-recording analysis.

BACKGROUND AND OBJECTIVES: Patients suffering from neurological diseases may develop dysarthria, a motor speech disorder affecting the execution of speech. Close and quantitative monitoring of dysarthria evolution is crucial for enabling clinicians to promptly implement patients' management strategies and maximizing effectiveness and efficiency of communication functions in term of restoring, compensating or adjusting. In the clinical assessment of orofacial structures and functions, at rest condition or during speech and non-speech movements, a qualitative evaluation is usually performed, throughout visual observation. METHODS: To overcome limitations posed by qualitative assessments, this work presents a store-and-forward self-service telemonitoring system that integrates, within its cloud architecture, a convolutional neural network (CNN) for analyzing video recordings acquired by individuals with dysarthria. This architecture - called facial landmark Mask RCNN - aims at locating facial landmarks as a prior for assessing the orofacial functions related to speech and examining dysarthria evolution in neurological diseases. RESULTS: When tested on the Toronto NeuroFace dataset, a publicly available annotated dataset of video recordings from patients with amyotrophic lateral sclerosis (ALS) and stroke, the proposed CNN achieved a normalized mean error equal to 1.79 on localizing the facial landmarks. We also tested our system in a real-life scenario on 11 bulbar-onset ALS subjects, obtaining promising outcomes in terms of facial landmark position estimation. DISCUSSION AND CONCLUSIONS: This preliminary study represents a relevant step towards the use of remote tools to support clinicians in monitoring the evolution of dysarthria.

Combining CNNs and Markov-like Models for Facial Landmark Detection with Spatial Consistency Estimates.

The accurate localization of facial landmarks is essential for several tasks, including face recognition, head pose estimation, facial region extraction, and emotion detection. Although the number of required landmarks is task-specific, models are typically trained on all available landmarks in the datasets, limiting efficiency. Furthermore, model performance is strongly influenced by scale-dependent local appearance information around landmarks and the global shape information generated by them. To account for this, we propose a lightweight hybrid model for facial landmark detection designed specifically for pupil region extraction. Our design combines a convolutional neural network (CNN) with a Markov random field (MRF)-like process trained on only 17 carefully selected landmarks. The advantage of our model is the ability to run different image scales on the same convolutional layers, resulting in a significant reduction in model size. In addition, we employ an approximation of the MRF that is run on a subset of landmarks to validate the spatial consistency of the generated shape. This validation process is performed against a learned conditional distribution, expressing the location of one landmark relative to its neighbor. Experimental results on popular facial landmark localization datasets such as 300 w, WFLW, and HELEN demonstrate the accuracy of our proposed model. Furthermore, our model achieves state-of-the-art performance on a well-defined robustness metric. In conclusion, the results demonstrate the ability of our lightweight model to filter out spatially inconsistent predictions, even with significantly fewer training landmarks.

Perception of smile attractiveness among laypeople and orthodontists regarding the buccal corridor space, as it is defined by the eyes. An innovated technique.

OBJECTIVE: To investigate whether there is a relationship between the distance between the iris and pupil with the ideal size of buccal corridors. MATERIALS AND METHODS: A full-portrait image of a male Caucasian was used to create a set of 11 digitally modified images with different buccal corridor space. A web-based cross-sectional study was designed and distributed via an online survey to 200 laypeople and 200 orthodontists to assess image attractiveness, using a Visual analogue scale. For the statistical analysis, Wilcoxon signed-rank and Mann-Whitney U tests were used. The significance level was set at p < 0.05. RESULTS: The response rate for laypeople was 70% (n = 139), while the rate for orthodontists was 73% (n = 146). For the layperson group, the maximum smile attractiveness score was 10% of buccal width reduction, compared to the iris-pupillary distance, while for the orthodontists, it was 20%. The attractiveness of the smile was significantly reduced in both groups when the buccal corridor width was increased in comparison to the iris-pupillary distance. CONCLUSION: The length between the mesial part of the iris and the distal of the pupil, may constitutes a landmark for the estimation of the desired width of the buccal corridor. CLINICAL RELEVANCE: Inter iris-pupillary distance can be the starting point in the smile designing process, in order to perform a facial driven selection of buccal corridor size.

Stacked attention hourglass network based robust facial landmark detection.

Deep learning based facial landmark detection (FLD) has made rapid progress. However, the accuracy and robustness of FLD algorithms are degraded heavily when the face is subject to diverse expressions, posture deflection, partial occlusion and other uncertain circumstances. To learn more discriminative representations and reduce the negative effect caused by outliers, a stacked attention hourglass network (SAHN) is proposed for FLD, where new attention mechanism is introduced. Basically, in the design of SAHN, a spatial attention residual (SAR) unit is constructed such that relevant areas of facial landmarks are specially emphasized and essential features of different scales can be well extracted, and a channel attention branch (CAB) is introduced to better guide the next-level hourglass network for feature extraction. Due to the introduction of SAR and CAB, only two hourglass networks are stacked as the proposed SAHN with fewer parameters, which is different from traditional SHNs stacked by four hourglass networks. Furthermore, a variable robustness (VR) loss function is introduced for the training of SAHN. The robustness of the proposed model for FLD is guaranteed with the help of the VR loss by adaptively adjusting a continuous parameter. Extensive experimental results on three public datasets including 300W, WFLW and COFW confirm that our method is superior to some previous ones.

Numerical Approach to Facial Palsy Using a Novel Registration Method with 3D Facial Landmark.

Treatment of facial palsy is essential because neglecting this disorder can lead to serious sequelae and further damage. For an objective evaluation and consistent rehabilitation training program of facial palsy patients, a clinician's evaluation must be simultaneously performed alongside quantitative evaluation. Recent research has evaluated facial palsy using 68 facial landmarks as features. However, facial palsy has numerous features, whereas existing studies use relatively few landmarks; moreover, they do not confirm the degree of improvement in the patient. In addition, as the face of a normal person is not perfectly symmetrical, it must be compared with previous images taken at a different time. Therefore, we introduce three methods to numerically approach measuring the degree of facial palsy after extracting 478 3D facial landmarks from 2D RGB images taken at different times. The proposed numerical approach performs registration to compare the same facial palsy patients at different times. We scale landmarks by performing scale matching before global registration. After scale matching, coarse registration is performed with global registration. Point-to-plane ICP is performed using the transformation matrix obtained from global registration as the initial matrix. After registration, the distance symmetry, angular symmetry, and amount of landmark movement are calculated for the left and right sides of the face. The degree of facial palsy at a certain point in time can be approached numerically and can be compared with the degree of palsy at other times. For the same facial expressions, the degree of facial palsy at different times can be measured through distance and angle symmetry. For different facial expressions, the simultaneous degree of facial palsy in the left and right sides can be compared through the amount of landmark movement. Through experiments, the proposed method was tested using the facial palsy patient database at different times. The experiments involved clinicians and confirmed that using the proposed numerical approach can help assess the progression of facial palsy.

Facial landmark-guided surface matching for image-to-patient registration with an RGB-D camera.

BACKGROUND: Fiducial marker-based image-to-patient registration is the most common way in image-guided neurosurgery, which is labour-intensive, time consuming, invasive and error prone. METHODS: We proposed a method of facial landmark-guided surface matching for image-to-patient registration using an RGB-D camera. Five facial landmarks are localised from preoperative magnetic resonance (MR) images using deep learning and RGB image using Adaboost with multi-scale block local binary patterns, respectively. The registration of two facial surface point clouds derived from MR images and RGB-D data is initialised by aligning these five landmarks and further refined by weighted iterative closest point algorithm. RESULTS: Phantom experiment results show the target registration error is less than 3 mm when the distance from the camera to the phantom is less than 1000 mm. The registration takes less than 10 s. CONCLUSIONS: The proposed method is comparable to the state-of-the-arts in terms of the accuracy yet more time-saving and non-invasive.

Deep-Learning-Based Stress Recognition with Spatial-Temporal Facial Information.

In recent times, as interest in stress control has increased, many studies on stress recognition have been conducted. Several studies have been based on physiological signals, but the disadvantage of this strategy is that it requires physiological-signal-acquisition devices. Another strategy employs facial-image-based stress-recognition methods, which do not require devices, but predominantly use handcrafted features. However, such features have low discriminating power. We propose a deep-learning-based stress-recognition method using facial images to address these challenges. Given that deep-learning methods require extensive data, we constructed a large-capacity image database for stress recognition. Furthermore, we used temporal attention, which assigns a high weight to frames that are highly related to stress, as well as spatial attention, which assigns a high weight to regions that are highly related to stress. By adding a network that inputs the facial landmark information closely related to stress, we supplemented the network that receives only facial images as the input. Experimental results on our newly constructed database indicated that the proposed method outperforms contemporary deep-learning-based recognition methods.

A Foreground-Aware Framework for Local Face Attribute Transfer.

In the context of social media, large amounts of headshot photos are taken everyday. Unfortunately, in addition to laborious editing and modification, creating a visually compelling photographic masterpiece for sharing requires advanced professional skills, which are difficult for ordinary Internet users. Though there are many algorithms automatically and globally transferring the style from one image to another, they fail to respect the semantics of the scene and are unable to allow users to merely transfer the attributes of one or two face organs in the foreground region leaving the background region unchanged. To overcome this problem, we developed a novel framework for semantically meaningful local face attribute transfer, which can flexibly transfer the local attribute of a face organ from the reference image to a semantically equivalent organ in the input image, while preserving the background. Our method involves warping the reference photo to match the shape, pose, location, and expression of the input image. The fusion of the warped reference image and input image is then taken as the initialized image for a neural style transfer algorithm. Our method achieves better performance in terms of inception score (3.81) and Fréchet inception distance (80.31), which is about 10% higher than those of competitors, indicating that our framework is capable of producing high-quality and photorealistic attribute transfer results. Both theoretical findings and experimental results are provided to demonstrate the efficacy of the proposed framework, reveal its superiority over other state-of-the-art alternatives.

Analysis of surgical outcome after upper eyelid surgery by computer vision algorithm using face and facial landmark detection.

PURPOSE: To evaluate the postoperative changes with a computer vision algorithm for anterior full-face photographs of patients who have undergone upper eyelid blepharoplasty surgery with, or without, a Müller's muscle-conjunctival resection (MMCR). METHODS: All patients who underwent upper eyelid blepharoplasty surgery (Group I), or upper eyelid blepharoplasty with MMCR (Group II) were included. Both preoperative and 6-month postoperative anterior full-face photographs of 55 patients were analyzed. Computer vision and image processing technologies were used to measure the palpebral distance (PD), eye-opening area (EA), and average eyebrow height (AEBH) for both eyes. Preoperative and postoperative measurements were calculated and compared between the two groups. RESULTS: In Group II, change in postoperative Right PD, Left PD, Right EA, Left EA was significantly higher than in Group I (p = 0.004 for REPD; p = 0.001 for LEPD; p = 0.004 for REA; p = 0.002 for LEA, p < 0.05). In Group II, the postoperative change in Right AEBH, Left AEBH was significantly higher than in Group I (p = 0.001 for RABH and LABH, p < 0.05). CONCLUSION: Eyelid surgery for esthetic purposes requires artistic judgment and objective evaluation. Because of the slight differences in photograph sizes and dynamic factors of the face due to head movements and facial expressions, it is hard to compare and make a truly objective evaluation of the eyelid operations. With a computer vision algorithm, using the face and facial landmark detection system, the photographs are normalized and calibrated. This system offers a simple, standardized, objective, and repeatable method of patient assessment. This can be the first step of Artificial Intelligence algorithm to evaluate the patients who had undergone eyelid operations.

Morphometric analysis of 3D soft-tissue for sexual dimorphism in human face / Análisis morfométrico de tejidos blandos 3D de dimorfismo sexual en rostro humano

Sexual dimorphism in Homo-sapiens is a phenomenon of a direct product of evolution by natural selection where evolutionary forces acted separately on the sexes which brought about the differences in appearance between male and female such as in shape and size. Advances in morphometrics have skyrocketed the rate of research on sex differences in human and other species. However, the current challenges facing 3D in the acquisition of facial data such as lack of homology, insufficient landmarks to characterize the facial shape and complex computational process for facial point digitization require further study in the domain of sex dimorphism. This study investigates sexual dimorphism in the human face with the application of Automatic Homologous Multi-points Warping (AHMW) for 3D facial landmark by building a template mesh as a reference object which is thereby applied to each of the target mesh on Stirling/ESRC dataset containing 101 subjects (male = 47, female = 54). The semi-landmarks are subjected to sliding along tangents to the curves and surfaces until the bending energy between a template and a target form is minimal. Principal Component Analysis (PCA) is used for feature selection and the features are classified using Linear Discriminant Analysis (LDA) with an accuracy of 99.01 % which demonstrates that the method is robust.

El dimorfismo sexual en el Homo-sapiens es un fenómeno directo de la evolución por selección natural, donde las fuerzas evolutivas actuaron por separado en los sexos, lo que provocó las diferencias en la apariencia entre hombres y mujeres, tal como la forma y tamaño. Los avances en el área de la morfometría, han generado un aumento significativo de las investigaciones en las diferencias de sexo en humanos y otras especies. Sin embargo, los desafíos actuales que enfrenta el 3D en el análisis de datos faciales, como la falta de homología, puntos de referencia insuficientes para caracterizar la forma facial y la complejidad del proceso computacional para la digitalización de puntos faciales, requiere un estudio adicional en el área del dimorfismo sexual. Este estudio investiga el dimorfismo sexual en el rostro humano con la aplicación de la deformación automática de múltiples puntos homólogos para el hito facial 3D, mediante la elaboración de una malla de plantilla como objeto de referencia, y se aplica en cada una de las mallas objetivas en el conjunto de datos Stirling / ESRC que contiene 101 sujetos (hombre = 47, mujer = 54). Los semi-puntos de referencia se deslizan a lo largo de las tangentes a las curvas y superficies hasta que la energía de flexión entre una plantilla y una forma objetivo es mínima. El análisis de componentes principales (PCA) se utiliza para la selección de características y las características se clasifican mediante el análisis discriminante lineal (ADL) con una precisión del 99,01 %, lo que demuestra la validez del método.

Landmark-based homologous multi-point warping approach to 3D facial recognition using multiple datasets.

Over the years, neuroscientists and psychophysicists have been asking whether data acquisition for facial analysis should be performed holistically or with local feature analysis. This has led to various advanced methods of face recognition being proposed, and especially techniques using facial landmarks. The current facial landmark methods in 3D involve a mathematically complex and time-consuming workflow involving semi-landmark sliding tasks. This paper proposes a homologous multi-point warping for 3D facial landmarking, which is verified experimentally on each of the target objects in a given dataset using 500 landmarks (16 anatomical fixed points and 484 sliding semi-landmarks). This is achieved by building a template mesh as a reference object and applying this template to each of the targets in three datasets using an artificial deformation approach. The semi-landmarks are subjected to sliding along tangents to the curves or surfaces until the bending energy between a template and a target form is minimal. The results indicate that our method can be used to investigate shape variation for multiple datasets when implemented on three databases (Stirling, FRGC and Bosphorus).

3-Dimensional facial expression recognition in human using multi-points warping.

BACKGROUND: Expression in H-sapiens plays a remarkable role when it comes to social communication. The identification of this expression by human beings is relatively easy and accurate. However, achieving the same result in 3D by machine remains a challenge in computer vision. This is due to the current challenges facing facial data acquisition in 3D; such as lack of homology and complex mathematical analysis for facial point digitization. This study proposes facial expression recognition in human with the application of Multi-points Warping for 3D facial landmark by building a template mesh as a reference object. This template mesh is thereby applied to each of the target mesh on Stirling/ESRC and Bosphorus datasets. The semi-landmarks are allowed to slide along tangents to the curves and surfaces until the bending energy between a template and a target form is minimal and localization error is assessed using Procrustes ANOVA. By using Principal Component Analysis (PCA) for feature selection, classification is done using Linear Discriminant Analysis (LDA). RESULT: The localization error is validated on the two datasets with superior performance over the state-of-the-art methods and variation in the expression is visualized using Principal Components (PCs). The deformations show various expression regions in the faces. The results indicate that Sad expression has the lowest recognition accuracy on both datasets. The classifier achieved a recognition accuracy of 99.58 and 99.32% on Stirling/ESRC and Bosphorus, respectively. CONCLUSION: The results demonstrate that the method is robust and in agreement with the state-of-the-art results.

Branched convolutional neural networks incorporated with Jacobian deep regression for facial landmark detection.

Facial landmark detection is to localize multiple facial key-points for a given facial image. While many methods have achieved remarkable performance in recent years, the accuracy remains unsatisfactory due to some uncontrolled conditions such as occlusion, head pose variations and illumination, under which, the L2 loss function is conventionally dominated by errors from those facial components on which the landmarks are hard predicted. In this paper, a novel branched convolutional neural network incorporated with Jacobian deep regression framework, hereafter referred to as BCNN-JDR, is proposed to solve the facial landmark detection problem. Our proposed framework consists of two parts: initialization stage and cascaded refinement stages. We firstly exploit branched convolutional neural networks as the robust initializer to estimate initial shape, which is incorporated with the knowledge of component-aware branches. By virtue of the component-aware branches mechanism, BCNN can effectively alleviate this issue of the imbalance errors among facial components and provide the robust initial face shape. Following the BCNN, a sequence of refinement stages are cascaded to fine-tune the initial shape within a narrow range. In each refinement stage, the local texture information is adopted to fit the facial local nonlinear variation. Moreover, our entire framework is jointly optimized via the Jacobian deep regression optimization strategy in an end-to-end manner. Jacobian deep regression optimization strategy has an ability to backward propagate the training error of the last stage to all previous stages, which implements a global optimization approach to our proposed framework. Experimental results on benchmark datasets demonstrate that the proposed BCNN-JDR is robust against uncontrolled conditions and outperforms the state-of-the-art approaches.

Facial expression recognition based on Electroencephalogram and facial landmark localization.

BACKGROUND: Facial expression recognition plays an essential role in affective computing, mental illness diagnosis and rehabilitation. Therefore, facial expression recognition has attracted more and more attention over the years. OBJECTIVE: The goal of this paper was to improve the accuracy of the Electroencephalogram (EEG)-based facial expression recognition. METHODS: In this paper, we proposed a fusion facial expression recognition method based on EEG and facial landmark localization. The EEG signal processing and facial landmark localization are the two key parts. The raw EEG signals is preprocessed by discrete wavelet transform (DWT). The energy feature vector is composed of energy features of the reconstructed signal. For facial landmark localization, images of the subjects' facial expression are processed by facial landmark localization, and the facial features are calculated by landmarks of essence. In this research, we fused the energy feature vector and facial feature vector, and classified the fusion feature vector with the support vector machine (SVM). RESULTS: From the experiments, we found that the accuracy of facial expression recognition was increased 4.16% by fusion method (86.94 ± 4.35%) than EEG-based facial expression recognition (82.78 ± 5.78%). CONCLUSION: The proposed method obtain a higher accuracy and a stronger generalization capability.

A single-camera gaze tracking system under natural light.

Gaze tracking is a human-computer interaction technology, and it has been widely studied in the academic and industrial fields. However, constrained by the performance of the specific sensors and algorithms, it has not been popularized for everyone. This paper proposes a single-camera gaze tracking system under natural light to enable its versatility. The iris center and anchor point are the most crucial factors for the accuracy of the system. The accurate iris center is detected by the simple active contour snakuscule, which is initialized by the prior knowledge of eye anatomical dimensions. After that, a novel anchor point is computed by the stable facial landmarks. Next, second-order mapping functions use the eye vectors and the head pose to estimate the points of regard. Finally, the gaze errors are improved by implementing a weight coefficient on the points of regard of the left and right eyes. The feature position of the iris center achieves an accuracy of 98.87% on the GI4E database when the normalized error is lower than 0.05. The accuracy of the gaze tracking method is superior to the-state-of-the-art appearance-based and feature- based methods on the EYEDIAP database.

A Comprehensive Performance Evaluation of Deformable Face Tracking "In-the-Wild".

Recently, technologies such as face detection, facial landmark localisation and face recognition and verification have matured enough to provide effective and efficient solutions for imagery captured under arbitrary conditions (referred to as "in-the-wild"). This is partially attributed to the fact that comprehensive "in-the-wild" benchmarks have been developed for face detection, landmark localisation and recognition/verification. A very important technology that has not been thoroughly evaluated yet is deformable face tracking "in-the-wild". Until now, the performance has mainly been assessed qualitatively by visually assessing the result of a deformable face tracking technology on short videos. In this paper, we perform the first, to the best of our knowledge, thorough evaluation of state-of-the-art deformable face tracking pipelines using the recently introduced 300 VW benchmark. We evaluate many different architectures focusing mainly on the task of on-line deformable face tracking. In particular, we compare the following general strategies: (a) generic face detection plus generic facial landmark localisation, (b) generic model free tracking plus generic facial landmark localisation, as well as (c) hybrid approaches using state-of-the-art face detection, model free tracking and facial landmark localisation technologies. Our evaluation reveals future avenues for further research on the topic.

Assessment for facial nerve paralysis based on facial asymmetry.

Facial nerve paralysis (FNP) is a loss of facial movement due to facial nerve damage, which will lead to significant physical pain and abnormal function in patients. Traditional FNP grading methods are solely based on clinician's judgment and are time-consuming and subjective. Hence, an accurate, quantitative and objective method of evaluating FNP is proposed for constructing a standard system, which will be an invaluable tool for clinicians who treat the patient with FNP. In this paper, we introduce a novel method for quantitative assessment of FNP which combines an effective facial landmark estimation (FLE) algorithm and facial asymmetrical feature (FAF) by processing facial movement image. The facial landmarks can be detected automatically and accurately using FLE. The FAF is based on the angle of key facial landmark connection and mirror degree of multiple regions on human face. Our method provides significant contribution as it describes the displacement of facial organ and the changes of facial organ exposure during performing facial movements. Experiments show that our method is effective, accurate and convenient in practice, which is beneficial to FNP diagnosis and personalized rehabilitation therapy for each patient.

Role for 2D image generated 3D face models in the rehabilitation of facial palsy.

The outcome for patients diagnosed with facial palsy has been shown to be linked to rehabilitation. Dense 3D morphable models have been shown within the computer vision to create accurate representations of human faces even from single 2D images. This has the potential to provide feedback to both the patient and medical expert dealing with the rehabilitation plan. It is proposed that a framework for the creation and measuring of patient facial movement consisting of a hybrid 2D facial landmark fitting technique which shows better accuracy in testing than current methods and 3D model fitting.